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Notes  for  Slide  2 


In  this  talk,  I  will  be  covering  the  two  aspects  of  Problem 
Formulation  and  Solution  Strategy,  which  are  covered  in  Chapters 
3  and  4  of  the  new  Code  of  Best  Practice. 

Many  contributors  across  NATO  involved  in  C2  assessment  have 
given  of  their  time  to  produce  these  chapters. 

They  represent  completely  new  material,  compared  to  the  previous 
version  of  the  NATO  Code  of  Best  Practice,  and  help  to  place  the 
analysis  of  such  complex  issues  within  a  broader  context. 
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CD 


Problem  Formulation 


ORGANIZATION 


•  “First  find  out  what  the  question  is  -  then  find 
out  what  the  real  question  is”  -  Vince  Roske 
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ORGANIZATION 

The  essence  of  Problem  Formulation  is  to  find  out  what  are  the  real 
issues  which  lie  behind  the  question  which  is  put  to  the  study  team. 
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Problem  Formulation  -  Why  Do  It? 


•  Effective  problem  formulation  is  fundamental 
to  the  success  of  all  analysis 

•  In  C2  assessment  the  problems  are  often  ill- 
defined  and  complex,  involving  many 
dimensions  and  a  rich  context 

•  The  problem  formulation  phase  should 
identify  the  Context  of  the  Study  and  aspects 
of  the  Problem  related  issues  -  the  what  and 
the  why 
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•  Effective  problem  formulation  is  fundamental  to  the  success  of  all 
analysis,  but  particularly  in  C2  assessment  because  the 
problems  are  often  ill-defined  and  complex,  involving  many 
dimensions  and  a  rich  context.  Problem  formulation  involves 
decomposition  of  the  analytic  problem  into  appropriate 
dimensions  such  as  structures,  functions,  mission  areas, 
command  echelons,  and  C2  systems. 

•  Within  the  whole  study  context,  we  are  attempting  at  this  first 
stage  to  understand  what  the  problem  really  is  and  why  it  is 
important  -  the  WHAT  and  WHY  of  the  study. 

•  The  problem  formulation  phase  should  identify  the  Context  of 
the  Study  and  aspects  of  the  Problem  related  issues. 
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Problem  Formulation  -  Context 


•  Geopolitical  context  that  bounds  the  problem 
space 

•  Aim  and  Objectives  of  the  analysis 

-  including  the  decisions  to  be  supported 

•  Generic  C2  issues 

•  Relevant  previous  studies 
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ORGANIZATION 

•  The  Context  of  the  Study  includes: 

•  Geopolitical  context  that  bounds  the  problem  space; 

-  political,  social,  historical,  economic,  geographic,  technological; 

-  actors; 

-  threats. 

•  Aim  and  Objectives  of  the  analysis; 

-  including  the  decisions  to  be  supported 

•  Generic  C2  issues; 

-  Generic  C2  issues  include  for  example,  key  systems,  doctrine, 
TTP,  organisational  structures,  and  key  assumptions  (e.g.  system 
performance  parameters) 

•  Relevant  previous  studies. 
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Problem  Formulation  -  Aspects  of  the 

Problem 


•  Issues  to  be  Addressed 

•  Assumptions 

•  High  Level  Measures  of  Merit  (Outputs) 

•  Independent  Variables  (Inputs)  (controllable 
and  uncontrollable) 

•  Constraints  on  the  Values  of  the  Variables 
(domain  and  range) 

-  The  problem  is  not  formulated  until  the 
Assessment  Team  has  specified  each  aspect 
of  the  problem. 
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ORGANIZATION 

The  Aspects  of  the  Problem  include: 

•  Issues  to  be  Addressed; 

•  Assumptions; 

•  High  Level  MoM; 

•  Independent  Variables  (controllable  and  uncontrollable);  and 

•  Constraints  on  the  Values  of  the  Variables  (domain  and 
range.) 

The  problem  is  not  formulated  until  the  Assessment  Team 
has  specified  each  aspect  of  the  problem. 
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In  Simple  Terms 


•  Begin  with  the  real  problem 

•  Identify  the  variables  bounding  the  problem 
space 

•  Determine  which  are  outputs  (dependent) 

•  Determine  which  are  inputs  (independent) 

•  Build  an  understanding  of  how  these  relate 

-A  voyage  of  discovery 
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ORGANIZATION 

In  simple  terms,  problem  formulation  can  thus  be  seen  as  an  iterative  process. 

First,  the  Team  must  identify  the  variables  that  bound  the  problem  space. 

Then  they  must  determine  which  of  these  are  outputs  (dependent  variables) 
and  which  of  these  are  inputs  (independent  variables). 

The  team  proceeds  by  iterating  to  build  an  understanding  of  how  these  relate 
to  each  other.  It  should  be  viewed  as  a  voyage  of  discovery.  In  most,  if  not  all 
cases  of  C2  assessment,  the  knowledge  domain  under  study  is  in  fact  a 
system  characterised  by  rich  interaction  and  feedback  among  all  the  factors  of 
variables  of  interest.  The  choice  of  dependent  variables  results  from  a  clear 
specification  of  the  issues  and  products  needed  to  satisfy  the  terms  of 
reference.  Independent  and  intervening  variables  are  also  chosen  based  on 
the  purpose  of  the  analysis. 

In  the  initial  problem  formulation  iteration,  it  is  critical  to  begin  with 
an  understanding  of  the  REAL  problem  rather  than  a  determination  to 
apply  readily  available  tools,  scenarios,  and  data. 
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Interaction  and  Feedback 


Problem  Formulation  Process 


Stakeholders 
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Assessment  Team 


Identifies  Key  Issues 
Characterizes  the  Context 
Identifies  ’’Real  ”  Issues 
Characterizes  Key  Elements 


Formulated  Problem 
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Context 

of  the  Study 


Problem 
Formulation 
Tools 
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This  shows  the  process  in  diagrammatic  terms  as  a  close  set  of 
interactions  between  the  stakeholders  (those  with  an  interest  in  the 
results  of  the  study),  the  sponsor  of  the  study  (with  a  direct  interest 
and  probably  also  responsible  for  funding  the  study)  and  the  study 
team. 

This  interaction  takes  into  account  the  political/military/economic 
etc.  contexts  of  the  study,  and  applies  appropriate  problem 
formulation  tools  to  help  in  the  analysis.  These  tools  will  be 
different  (in  general)  from  those  ‘solving’  models  and  tools  used 
later  in  the  study  process. 
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Principles  of  Problem  Formulation 


•  Do  it  before  developing  concepts  for  analysis  or 
model  selection 

•  Understand  the  decisions  to  be  supported  and 
stakeholder  viewpoints 

•  Carefully  review  previous  work 

•  Do  analysis  and  synthesis 

•  Be  broad  and  iterative 

•  Practical  constraints  are  modifiers  not  drivers 

•  Address  risks  to  the  study  explicitly 
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Explicit  problem  formulation  must  precede  construction  of  concepts  for  analysis  or  method  selection. 
Especially  in  C2  assessments,  this  is  not  a  trivial  exercise.  Proper  resourcing  is  essential. 

An  understanding  of  the  decisions  to  be  supported  by  the  analysis  and  the  viewpoints  of  the  various 
stakeholders  (e.g.,  customers,  users,  and  suppliers)  is  essential  to  clarifying  the  study  issues. 

A  careful  review  of  previous  work  must  be  carried  out  as  a  valuable  source  of  ideas,  information,  and 
insight. 

Problem  formulation  must  not  only  analyse  (dissect),  but  also  provide  a  clear  and  valid  mechanism  for 
meaningful  synthesis  to  yield  coherent  knowledge  about  the  original,  larger  problem. 

It  must  be  broad  and  iterative  in  nature,  accepting  the  minimum  of  a  priori  constraints  and  using 
methods  to  encourage  creative  and  multi-disciplinary  thinking.  Change  is  inevitable  in  many  dimensions 
Thus  the  assessment  process  must  anticipate  this  change  and  accommodate  it. 

Practical  constraints  such  as  data  availability,  study  resources  (including  time)  and  limitations  of  tools 
should  be  treated  as  modifiers  on  the  problem  formulation  rather  than  initial  drivers.  Problem 
Formulation  should  address  risk  from  multiple  perspectives.  In  addition  to  sensitivity  analysis  of  the 
dependent  variables,  risk  analysis  techniques  should  be  used  to  directly  explore  options  to  mitigate  risk. 

Proper  Problem  Formulation  takes  substantial  time  and  effort! 
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Additional  Principles  for  OOTW  C2 

Assessment 


•  Address  the  Geopolitical  Context  and  generic 
C2  issues 

•  Look  at  Policy  level  impacts  -  MoPE 

•  History  can  be  important 

•  Do  not  focus  in  too  soon 

A  broad  range  of  disciplines  (e.g.  social 
scientists,  historians,  and  regional  experts 
in  OOTW  assessment)  is  required 


PR1-17 


Notes  for  Slide  18 


Problem  formulation  must  address  the  geopolitical  context  of  the  problem  and  seek  to 
identify  the  “generic”  C2  issues  contained  within  the  terms  of  reference  for  the  study. 

OOTW  C2  assessments  often  involve  policy-related  impacts  outside  the  context  of  a 
particular  military  operation.  Therefore,  MoM  hierarchies  must  contain  measures  of  policy 
effectiveness. 

An  historical  perspective  is  critical  to  understanding  OOTW  because  social  conflict,  and 
structures  often  have  roots  far  back  in  history.  However,  it  must  be  remembered  that 
present-day  social  behaviour  is  not  driven  by  historical  events  themselves,  but  by 
present-day  perceptions,  processes,  and  prejudices  which  have  evolved  from  the  past. 

A  key  risk  in  complex  OOTW  studies  is  allowing  the  problem  formulation  process  to  focus 
in  prematurely  on  subsets  of  the  problem  because  they  are:  a)  interesting;  b)  familiar; 
c)  pre-judged  to  be  critical;  d)  explicitly  called  out  by  the  customer  or  e)  easy  to  analyse. 
This  requires  great  discipline  by  the  study  team,  especially  where  the  team’s  previous 
experience  is  biased  in  favour  of  particular  parts  of  the  problem  space. 

The  Assessment  Team  needs  access  to  subject  matter  experts  from  a  broad  range 
of  disciplines  (e.g.  social  scientists ,  historians,  and  regional  experts  in  OOTW 
assessment.) 
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Chains  of  Cause  and  Effect 


ORGANIZATION 
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During  the  early  stages  of  Problem  Formulation  it  is  important  to 
quickly  cover  the  whole  problem  and  produce  an  initial  formulation 
(i.e.,  an  explicit  expression  of  the  problem).  This  prevents 
premature  narrowing  of  the  assessment  and  serves  as  an  aid  to 
shared  situation  awareness  within  the  study  team.  A  good 
approach  is  to  use  influence  diagrams  such  as  the  one  shown  here. 

In  formulating  an  OOTW  problem  we  are  trying  to  bound  a  complex 
system.  This  is  partly  a  process  of  understanding  boundaries  which 
exist  in  reality  (such  as  Mission  Statements  and  Geographical 
Areas)  and  partly  imposing  artificial  boundaries  in  order  to 
illuminate  the  structure  of  the  problem  and  constrain  the  scope  of 
the  analysis. 


PR1-20 


PR1-21 


Notes  for  Slide  22 


In  dealing  with  fuzzy  or  uncertain  boundaries,  the  problem 
formulation  process  needs  to  explore  and  understand  the 
significance  of  each  boundary  before  making  (or  seeking  from 
customers)  assumptions  about  it.  This  involves  keeping  an  open 
mind,  during  the  early  stages  of  problem  formulation,  about  where 
the  boundaries  lie  and  their  dimensional  nature.  This  is  difficult 
because  it  makes  the  problem  modelling  process  more 
complicated.  A  call  for  hard  specification  too  early  in  the  problem 
formulation  process  must  be  avoided.  In  the  end,  of  course,  the 
problem  must  be  formulated  in  order  to  solve  it,  but  formulation 
should  be  an  output  from  the  first  full  iteration,  not  an  early  input  to 
it.  The  danger  is  that  we  exclude  dimensions  from  the  analysis 
which  turn  out  to  be  important  later. 
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Bounding  the  Problem 
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Boundaries  (especially  self-imposed  ones)  should  be  kept  porous, 
allowing  for  cause  and  effect  chains  as  shown  here,  to  flow  through 
the  external  environment  of  the  (partial)  complex  system  that  the 
boundaries  define. 

Problems  where  the  chains  of  cause  and  effect  go  outside  the 
boundary  and  then  back  in  are  particularly  difficult  to  cope  with. 

While  clear  definitions  and  hard  conceptual  boundaries  are 
ultimately  necessary  in  order  to  constrain  the  problem  space 
into  a  manageable  form ,  care  must  be  taken  to  avoid  coming 
to  closure  prematurely. 


PR1-24 


Development  of  High  Level  MoM 


ORGANIZATION 
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Identification  of  high  level  MoM  should  start  with  ideal  measures  of 
the  desired  benefits  before  considering  what  can  be  practically 
generated  by  analysis  (the  latter  may  force  the  use  of  surrogate 
MoM,  but  these  must  be  clearly  related  to  the  desired  measures). 

A  structured  analysis  of  potential  benefits  should  be  carried  out  as 
a  basis  for  constructing  appropriate  Measures  of  Merit  (MoM). 
Mapping  techniques  (  such  as  cognitive  and  causal  mapping)  are  a 
good  way  to  express  the  various  relationships  within  the  problem 
space  and  to  identify  ‘chains’  of  influence  (as  shown  here).  These 
lead  on  to  resultant  structure  in  terms  of  independent  and 
dependent  variables,  and  hence  to  high  level  MoM. 
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A  Structured  Analysis  of 
Measures  of  Merit 
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The  structured  analysis  of  benefits  is  a  logical  process  that  seeks 
causally  to  map  lower  level  Measures  of  Merit  that  can  be  related  to 
investments  or  other  actions  to  higher  level  Measures  that  can  be 
valued  directly  by  decision-makers. 

As  shown  here,  we  can  consider  such  measures  of  Merit  as 
forming  a  set  of  expanding  circles  going  from  the  C2  subsystem  to 
the  Force  and  the  environment. 

This  corresponds  to  a  hierarchy  of  Measures  of  Merit  ranging  from 
Performance  through  C2  Effectiveness  to  Force  Effectiveness  and 
Policy  Effectiveness. 
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A  Structured  Analysis  of 
Measures  of  Merit 


ORGANIZATION 
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In  the  initial  problem  formulation,  we  should  focus  initially  on  the 
higher  level  end  of  this  set  of  Measures  of  Merit,  with  the  more 
detailed  Measures  being  considered  later  in  the  Study  process. 
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Tools  for  Problem  Formulation 


•  Elicitation  support 

•  Influence  diagrams 

•  Causal  maps 

•  System  dynamic  models 

•  Agent  based  ‘distillations’ 

•  Etc 


PR1-31 


Notes  for  Slide  32 


It  is  useful  to  identify,  develop  if  necessary,  and  apply  appropriate 
tools  to  support  Problem  Formulation.  Representative  tools 
and/techniques  include  those  listed  here:  techniques  for  supporting 
expert  elicitation,  influence  diagrams,  causal  maps,  system 
dynamic  models,  and  agent-based  models. 

Tools  and  approaches  used  for  Problem  Formulation  must  be 
coherent  with  other  tools  and  techniques  likely  to  be  considered  for 
the  subsequent  analysis,  in  order  to  produce  a  sensible 
‘multimethodology’  approach  to  the  entire  problem  and  its  solution. 

This  means  a  set  of  approaches  which  fit  together  across  the  span 
of  the  study  process. 
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The  Formulated  Problem 
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ORGANIZATION 

The  formulation  of  the  problem  is  completed  when  the  constraints  on  either 
the  independent  or  dependent  variables  have  been  identified.  Constraints 
on  the  dependent  variables  represent  “acceptable”  thresholds  or  limits.  For 
example,  one  could  place  a  constraint  on  blue  losses,  time  to  accomplish  a 
mission,  collateral  damage,  or  some  combination  of  factors.  Constraints  on 
the  independent  variables  represent  either  feasible  or  acceptable  limits  on 
such  factors  as  human  performance,  C2  system  performance,  or  even 
supplies.  They  also  could  represent  doctrinal  or  legal  processes  that  act  as 
constraints. 

The  next  step  in  the  C2  assessment  process  is  the  development  of  a 
solution  strategy.  It  should  be  noted  that  the  Team  is  not  finished  with 
Problem  Formulation  at  this  point,  but  is  now  ready  to  proceed.  As  work 
progresses  on  the  development  of  a  Solution  Strategy,  it  will  also  certainly 
be  necessary  to  revisit  the  specification  of  high-level  MoMs  and  the 
constraints. 
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This  leads  us  on  then  to  the  Solution  Strategy. 

We  will  look  at  this  from  the  point  of  view  of  developing  a  plan  for 
the  study,  which  takes  account  of  the  problem  formulation  we  have 
just  been  discussing. 
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The  Study  Plan 


•  The  what 

-  the  formulated  problem 

•  The  how 

-  the  solution  strategy 
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It  is  important  that  there  be  a  conscious  effort  to  create  and  follow  a  study 
plan  that  guides  data  collection  and  analyses  and  is  prepared  to  use  the 
insights  and  data  that  are  collected  to  create  a  solution  to  the  problem  at 
hand.  The  study  plan  consists  of  two  inter-related  parts  -  the  formulated 
problem  (the  What)  and  the  solution  strategy  (the  How). 

The  output  of  the  initial  Problem  Formulation  provides  the  Team  with  an 
operating  definition  of  what  needs  to  be  done. 

The  output  of  the  Solution  Strategies  phase  provides  the  Team  with  an 
initial  understanding  of  how  this  will  be  accomplished.  As  the  project 
unfolds,  there  will  usually  be  a  significant  amount  of  iteration  that  both 
modifies  the  problem  formulation  and  the  solution  strategy.  Without  a 
study  plan,  it  is  unlikely  that  efforts  will  be  properly  scoped,  prioritised, 
scheduled,  and  resourced. 

Even  if  the  way  ahead  seems  clear,  the  articulation  of  a  formal 
Solution  Strategy  is  necessary. 
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From  Formulation  to  Solution 


Study 

Constraints 
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We  can  see  here  what  is  involved  in  moving  from  a  formulation 
problem  to  a  solution  strategy.  The  objective  of  this  phase  of  the 
study  is  to  develop  a  feasible  approach  to  go  from  the  specification 
of  what  is  to  be  done  to  how  it  is  to  be  done. 

This  involves  developing  an  approach  that  will  result  in  the  Team’s 
ability  to  collect  the  data  necessary  to  determine  the  values  of  the 
Measures  of  Merit  for  specified  values  of  independent  variables. 

The  kinds  of  data  collection  instruments  and  analysis  tools  and 
techniques  used  will  determine  the  resources  required,  the  time 
needed,  and  the  risks  inherent  in  the  solution  approach.  At  all  times 
we  need  to  bear  in  mind  the  study  constraints  (time  and  resources 
available)  and  the  key  problem  issues  (e.g.  are  we  measuring  the 
right  thing?). 
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Study  Constraints  and  Factors 


•  Measures  of  Merit 

•  Human  Issues 

•  Scenarios 

•  Methods  and  Tools 

•  Data 

•  Risk  and  Uncertainty 
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In  considering  the  strategy  for  the  study,  these  are  the  key  factors 
and  constraints  which  need  to  be  considered.  They  form  separate 
Chapters  of  the  new  Code  of  Best  Practice,  and  will  be  discussed 
in  detail  by  other  speakers,  so  I  will  not  dwell  further  on  them  here. 
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Steps  in  Developing 
a  Solution  Strategy 
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As  an  initial  step,  the  development  of  the  set  of  MoM  to  be  used  in  the  study 
anchors  the  process  that  will  eventually  lead  to  a  solution  strategy. 

This  process  revolves  around  the  conceptual  model  that  the  Assessment  Team 
builds.  It  is  best  practice  to  make  this  model  explicit  and  have  it  serve  as  the 
common  picture  that  develops  a  high  quality  of  shared  understanding  among  the 
Team,  Sponsors,  Stakeholders,  and  other  key  study  participants.  The  initial 
conceptual  model  consists  simply  of  the  MoMs,  a  first  cut  of  the  hypothesised 
relationships  among  them,  assumptions  about  variables  and  their  relationships,  and 
constraints.  Later  iterations  include  additional  independent  variables  that  are  known 
or  assumed  to  affect  the  values  of  the  MoM  or  the  nature  of  the  relationship  among 
them,  and  increasingly  detailed  specifications  of  relationships,  and  specific  values  or 
ranges  for  the  independent  variables. 

Taken  together,  the  detailed  specification  of  the  MoM,  the  development  of  a 
conceptual  model  including  the  relevant  human  and  organisational  factors, 
the  specification  of  a  set  of  scenarios,  and  a  data  collection  and  analysis  plan 
(that  consists  of  the  methods  and  tools  to  be  used)  constitutes  a  solution  strategy. 
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Iterative  Development  of  the 

Study  Plan 


PR1-45 


Notes  for  Slide  46 


We  can  see  here  the  iterative  nature  of  the  process  involved  in 
developing  the  Overall  Study  Plan. 

A  first  order  feed  back  loop  is  shown  between  Problem  Formulation 
and  Solution  Strategy,  with  both  processes  having  iterative  internal 
processes.  An  analysis  of  Risk  and  Uncertainty  provides  the  control 
mechanism  that  drives  the  iteration  to  an  acceptable  result. 
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Study  Management  Plan 


•  Detailed  guidance  of  the  project 

•  Time  phased  execution  plan 

•  Work  Breakdown  Structure 

•  Delivery  Milestones 

•  Etc 
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In  addition  to  the  Study  Plan,  the  Team  should  also  create  and 
maintain  a  detailed  Study  Management  Plan 

This  should  cover  the  kind  of  issues  described  above. 

Together  with  the  broad  Study  Plan,  this  will  help  ensure  that  the 
study  is  of  high  technical  quality,  and  delivers  the  right  results  at  the 
right  time. 
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Final  Thoughts 


•  Effort  spent  ‘up  front’  in  formulating  the 
problem  will  pay  off  later 

•  It  is  better  to  be  approximately  right  than 
precisely  wrong 

•  ‘A  theory  should  be  a  simple  as  possible  -  but 
no  simpler’  (Albert  Einstein) 
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Spending  more  time  in  the  initial  part  of  the  study  on  exploring  the 
nature  of  the  problem  pays  off  later. 

Firstly  the  study  will  address  directly  the  key  issues  of  concern  to 
the  decision  makers.  This  creates  an  atmosphere  of  trust  and 
engagement  which  helps  in  implementing  the  results  of  the  study. 

Secondly,  the  precise  answer  to  the  wrong  problem  will  have  no 
impact.  This  is  the  danger  of  narrowing  the  study  down  too  quickly 
to  those  aspects  which  are  easy  to  do. 

Finally,  I  started  with  a  quote,  so  I  shall  end  with  one.  In  our  terms  it 
can  be  expressed  as:  avoid  unnecessary  complication.  Sometimes, 
simply  formulating  the  problem  in  a  way  which  the  decision  maker 
can  full  grasp  is  a  significant  step  forward. 
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